GitHub Actionsでコンテナ展開時にno space left on deviceで失敗する場合の暫定対応

はじめに

GitHub ActionsでGlue用のコンテナを使ってユニットテスト(midium)を実行しています。詳しい構成はこちらの記事に記載しています。

リポジトリテンプレートはこちらです。今回このリポジトリで起きた事象ではなく、forkしたプロダクトで突然発生した事象です。

あくまで暫定対応で、急にディスク容量が足りなくなった理由は不明です。

事象

コンテナをpullしているときに、以下のエラーがGitHub Actions上で出るようになりました。

glue_libs_4.0.0_image_01: Pulling from amazon/aws-glue-libs
(中略)
08ae846c7970: Pull complete
1d40ee859868: Pull complete
Digest: sha256:1bb354fe4652e1864cdb01a88f2fb19fae5311e152504fbec5f91e3f7e635ba3
Status: Downloaded newer image for amazon/aws-glue-libs:glue_libs_4.0.0_image_01
docker.io/amazon/aws-glue-libs:glue_libs_4.0.0_image_01
write ./.docker_temp_3918240345: no space left on device

エラーメッセージの通り、コンテナを展開する空き容量がないと推測しました。

問題が起こったworkflowのファイルは以下の通りです。

name: CI src-glue

on:
  push:
    paths:
      - 'packages/src-glue/**'
      - '.github/workflows/ci-glue.yml'

env:
  GLUE_LIB_CACHE_PATH: glue-libs-image
  GLUE_LIB_VERSION: 4.0.0

jobs:
  ci:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Cache a GlueLib Docker image
        id: cache-python-packages
        uses: actions/cache@v4
        with:
          path: ${{ env.GLUE_LIB_CACHE_PATH }}
          key: ${{ runner.os }}-glue-libs-${{ env.GLUE_LIB_VERSION }}

      - name: Pull and save a GlueLib Docker image
        if: steps.cache-python-packages.outputs.cache-hit != 'true'
        run: |
          docker pull amazon/aws-glue-libs:glue_libs_${GLUE_LIB_VERSION}_image_01
          docker save amazon/aws-glue-libs:glue_libs_${GLUE_LIB_VERSION}_image_01 -o ${GLUE_LIB_CACHE_PATH}

      - name: lunch container
        run: |
          cd packages/src-glue/
          docker-compose up -d

      - name: Install Rye
        run: |
          ./setup-rye.sh

      - name: Restore python packages
        uses: actions/cache@v4
        id: cache_dependency_python
        env:
          cache-name: cache-dependency-python
        with:
          path: '.venv'
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('requirements-dev.lock') }}

      - name: Install python module
        if: ${{ steps.cache_dependency_python.outputs.cache-hit != 'true' }}
        run: |
          source "$HOME/.rye/env"
          rye sync

      - uses: actions/setup-node@v4
        with:
          node-version-file: ./.node-version

      - name: Restore node modules
        uses: actions/cache@v4
        id: cache_dependency_node
        env:
          cache-name: cache-dependency-node
        with:
          path: '**/node_modules'
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('package-lock.json') }}

      - name: Install node modules
        if: ${{ steps.cache_dependency_node.outputs.cache-hit != 'true' }}
        run: npm ci --no-audit --progress=false --silent

      - name: CI
        shell: bash
        run: |
          source "$HOME/.rye/env"
          npm run check \
            -w @glue-template/src-glue
          npm run test \
            -w @glue-template/src-glue

因みにamazon/aws-glue-libsのサイズを調べて見たところ6GB程度あることがわかりました。

$ docker system df -v
Images space usage:
REPOSITORY             TAG                        IMAGE ID       CREATED        SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
amazon/aws-glue-libs   glue_libs_4.0.0_image_01   92af3491c369   4 months ago   6.02GB    0B            6.02GB        0
Containers space usage:
CONTAINER ID   IMAGE     COMMAND   LOCAL VOLUMES   SIZE      CREATED   STATUS    NAMES
Local Volumes space usage:
VOLUME NAME   LINKS     SIZE
Build cache usage: 0B
CACHE ID   CACHE TYPE   SIZE      CREATED   LAST USED   USAGE     SHARED

解消法

調べるといくつか方法はありそうでした。

1. ホストの利用しない(コンテナを除く)ファイルを削除する
2. コンテナのデフォルトイメージを削除する
3. サードパーティアクションjlumbroso/free-disk-spaceを使う

個人的な問題で恐縮ですが、3はGHEで署名済みのサードパーティアクションしか使えないという制約があり、2.のGitHub Actions上にデフォルトで入っているコンテナイメージを削除することにしました。まずどの程度デフォルトでイメージが入っているかを確認します。

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          14        0         3.148GB   3.148GB (100%)
Containers      0         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B

$ docker system df -v
Images space usage:

REPOSITORY      TAG         IMAGE ID       CREATED        SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
moby/buildkit   latest      a86a00f5eb13   2 weeks ago    198MB     7.372MB       190.7MB       0
node            18-alpine   995e68c9d946   2 weeks ago    132MB     7.372MB       124.3MB       0
node            20-alpine   ae7f6c370720   2 weeks ago    137MB     7.372MB       129.3MB       0
node            18          9489b43a1afc   3 weeks ago    1.09GB    928.8MB       163.7MB       0
node            20          2e805f601f2b   3 weeks ago    1.1GB     928.8MB       168.1MB       0
debian          10          1048ff9a62c6   3 weeks ago    114MB     0B            114.2MB       0
debian          11          b4210c79b3c3   3 weeks ago    124MB     0B            124.2MB       0
ubuntu          22.04       ca2b0f26964c   5 weeks ago    77.9MB    0B            77.86MB       0
ubuntu          20.04       3cff1c6ff37e   6 weeks ago    72.8MB    0B            72.8MB        0
alpine          3.16        d49a5025be10   2 months ago   5.54MB    0B            5.54MB        0
alpine          3.17        eaba187917cc   2 months ago   7.06MB    0B            7.058MB       0
alpine          3.18        d3782b16ccc9   2 months ago   7.33MB    0B            7.335MB       0
node            16          1ddc7e4055fd   7 months ago   909MB     0B            908.8MB       0
node            16-alpine   2573171e0124   7 months ago   118MB     0B            117.5MB       0

Containers space usage:

CONTAINER ID   IMAGE     COMMAND   LOCAL VOLUMES   SIZE      CREATED   STATUS    NAMES

Local Volumes space usage:

VOLUME NAME   LINKS     SIZE

Build cache usage: 0B

CACHE ID   CACHE TYPE   SIZE      CREATED   LAST USED   USAGE     SHARED

3G程度入っていることがわかりました。

修正後の完成形はこちらです。

name: CI src-glue

on:
  push:
    paths:
      - 'packages/src-glue/**'
      - '.github/workflows/ci-glue.yml'

env:
  GLUE_LIB_CACHE_PATH: glue-libs-image
  GLUE_LIB_VERSION: 4.0.0

jobs:
  ci:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

+     - name: Adjutment docker volumes
+       run: |
+         docker system df
+         docker system prune -a --volumes -f
+         docker system df

      - name: Cache a GlueLib Docker image
        id: cache-python-packages
        uses: actions/cache@v4
        with:
          path: ${{ env.GLUE_LIB_CACHE_PATH }}
          key: ${{ runner.os }}-glue-libs-${{ env.GLUE_LIB_VERSION }}

      - name: Pull and save a GlueLib Docker image
        if: steps.cache-python-packages.outputs.cache-hit != 'true'
        run: |
          docker pull amazon/aws-glue-libs:glue_libs_${GLUE_LIB_VERSION}_image_01
          docker save amazon/aws-glue-libs:glue_libs_${GLUE_LIB_VERSION}_image_01 -o ${GLUE_LIB_CACHE_PATH}

      - name: lunch container
        run: |
          cd packages/src-glue/
          docker-compose up -d

      - name: Install Rye
        run: |
          ./setup-rye.sh

      - name: Restore python packages
        uses: actions/cache@v4
        id: cache_dependency_python
        env:
          cache-name: cache-dependency-python
        with:
          path: '.venv'
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('requirements-dev.lock') }}

      - name: Install python module
        if: ${{ steps.cache_dependency_python.outputs.cache-hit != 'true' }}
        run: |
          source "$HOME/.rye/env"
          rye sync

      - uses: actions/setup-node@v4
        with:
          node-version-file: ./.node-version

      - name: Restore node modules
        uses: actions/cache@v4
        id: cache_dependency_node
        env:
          cache-name: cache-dependency-node
        with:
          path: '**/node_modules'
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('package-lock.json') }}

      - name: Install node modules
        if: ${{ steps.cache_dependency_node.outputs.cache-hit != 'true' }}
        run: npm ci --no-audit --progress=false --silent

      - name: CI
        shell: bash
        run: |
          source "$HOME/.rye/env"
          npm run check \
            -w @glue-template/src-glue
          npm run test \
            -w @glue-template/src-glue

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          14        0         3.148GB   3.148GB (100%)
Containers      0         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B
Deleted Images:
untagged: node:18
untagged: node@sha256:b39895225fb1984139d5af76400aff8fac3dd5bc00dd41a3ce22fc8a6cf538d5
deleted: sha256:9489b43a1afc6909c366b4ddd9e448da2cc54de8c9705f3bf219489f2e7e9e50
deleted: sha256:797d7d927834e3d38642b7979b7b083ee7c11a166b2d7a1d2640b29ace219704
deleted: sha256:b8fe4396a0bf133807f542947a12e0e60815d15f34f8c6f0e42e60afced4d752
deleted: sha256:febeec8a54f9ff594226e53838b52590da58b5cde7edf2f096037393194c75f3
untagged: node:20-alpine
untagged: node@sha256:bf77dc26e48ea95fca9d1aceb5acfa69d2e546b765ec2abfb502975f1a2d4def
deleted: sha256:ae7f6c370720be0fd4120606ee84f6170037f6c163b91522eea08cbe574d78ff
deleted: sha256:912c5cec1262e60658f609ad96745100a6e67f29db40d751c3401e954964a0ab
deleted: sha256:ac26ab7a4bf0495172054353a303c5589c17151342d745e6f57079ee34c9d83a
deleted: sha256:064c6bbdbbec7b1d1fd2261055fc5e44ace2fb3eabe76c3a1e2dabcead0b2c03
untagged: moby/buildkit:latest
untagged: moby/buildkit@sha256:00d2c6b8f39ae515e0eadd74f39e71a5efdc94321c9b919692a2aa32deef2bb1
deleted: sha256:a86a00f5eb13d4b7290619228353064eaed37cc31b42f9f9228dfcb5a904f0dc
deleted: sha256:601175bc8ddfe19d6ed4e5441c85bf1e3002deb88904d2d8a1ecbed37a19aad4
deleted: sha256:36dcd0b1151e77e8b9389fdb8207fc665cdc3a9507fcc4acf27b5416b8b6a0ca
deleted: sha256:7f38fa97ceb1fa4bef1d50bce45b7a505ac545fd073295f00e30abc36a5bd18a
untagged: ubuntu:22.04
untagged: ubuntu@sha256:77906da86b60585ce12215807090eb327e7386c8fafb5402369e421f44eff17e
deleted: sha256:ca2b0f26964cf2e80ba3e084d5983dab293fdb87485dc6445f3f7bbfc89d7459
deleted: sha256:5498e8c22f6996f25ef193ee58617d5b37e2a96decf22e72de13c3b34e147591
untagged: alpine:3.17
untagged: alpine@sha256:53cf9478b76f4c8fae126acbdfb79bed6e69e628faff572ebe4a029d3d247d98
deleted: sha256:eaba187917ccc6af1386ecfdd95a56858a553d14888387980fb07b326e8ce890
deleted: sha256:f4111324080ce5b633fab04c0f3f21b587f2ac10a289cc9e2760c67e0d26711c
untagged: node:16-alpine
untagged: node@sha256:a1f9d027912b58a7c75be7716c97cfbc6d3099f3a97ed84aa490be9dee20e787
deleted: sha256:2573171e0124bb95d14d128728a52a97bb917ef45d7c4fa8cfe76bc44aa78b73
deleted: sha256:dc17c06667c0136a5cf7f0b276880355138f030a23fb38cc30875003d3ca8923
deleted: sha256:346861fde1f6e6b9b51e4fe1442afef18f3dfc9aaf65ee3fd0802f7d4bd3e14e
deleted: sha256:a3e01e6e4f9ab3d47cd8a57272302179a1f0d90b7d7dab5d4572065dc0ac2aeb
deleted: sha256:4693057ce2364720d39e57e85a5b8e0bd9ac3573716237736d6470ec5b7b7230
untagged: alpine:3.16
untagged: alpine@sha256:452e7292acee0ee16c332324d7de05fa2c99f9994ecc9f0779c602916a672ae4
deleted: sha256:d49a5025be10344cce77d178103a225cb5d7316861e5d8f106e7ff278ae51b62
deleted: sha256:5535fda0356bb3eff348b2f618314a47946d7ccfbeb880b3fc910dd7375ae140
untagged: ubuntu:20.04
untagged: ubuntu@sha256:80ef4a44043dec4490506e6cc4289eeda2d106a70148b74b5ae91ee670e9c35d
deleted: sha256:3cff1c6ff37e0101a08119d0743ba95c7f505d00298f5a169129e4e9086cde9e
deleted: sha256:5faf9c0a9efe4675ecd21a4ec417d51077d5e75da9e673161a94e7d6cd43f92c
untagged: debian:11
untagged: debian@sha256:5a87974e73c64b3fb161d444a84bdd47c0e6b6058eacaeea64342e7cbce1f04d
deleted: sha256:b4210c79b3c32eb6979f1643346545cb0f90190d6509f8b3706dc7e0c0c43293
deleted: sha256:ac2949c6975728f37ddee46b942f29429780aeb268b0116d65dc95346d5566b6
untagged: node:18-alpine
untagged: node@sha256:c7620fdecfefb96813da62519897808775230386f4c8482e972e37b8b18cb460
deleted: sha256:995e68c9d946f454675dd18204619ccd3b8e065f55f0ae1996f00300722554f9
deleted: sha256:f9c0cd2629ac29b367b2b874cb5c6e12c9075286849d9f78fb2f9daf0e1e0c61
deleted: sha256:206e9cb8596a67deadf0d9091935a86f5f9868c6b36f2888f06ae6333c9e738b
deleted: sha256:22016d011f54ffec96f73405756d4b00fd01294d91aee583dd1b342e1e0979f7
deleted: sha256:d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820
untagged: debian:10
untagged: debian@sha256:f6b3b7c7b049c2c7d0f19ae988b4eac64fd8e127fa891c9de1d3cf3f8c33cad4
deleted: sha256:1048ff9a62c68759bfea343a5e6039b909f35107896d5d2ced130d8353464f04
deleted: sha256:d34336bf18eebcd9ae0360169fad8a3740b611741d3289335a4d0c3e6c91b33a
untagged: alpine:3.18
untagged: alpine@sha256:11e21d7b981a59554b3f822c49f6e9f57b6068bb74f49c4cd5cc4c663c7e5160
deleted: sha256:d3782b16ccc94322a5c5a7d004192b5daa2a1ecd61c143074e36dba844408e1c
deleted: sha256:aedc3bda2944bb9bcb6c3d475bee8b460db9a9b0f3e0b33a6ed2fd1ae0f1d445
untagged: node:20
untagged: node@sha256:e06aae17c40c7a6b5296ca6f942a02e6737ae61bbbf3e2158624bb0f887991b5
deleted: sha256:2e805f601f2baab256bea06f32b5bf1e0b5f3dfc6e4287d67368553be34b39fe
deleted: sha256:4217246cf1fd34d58e1eee5fd9f7278663911867a3f399b1a76c013e9433c6aa
deleted: sha256:a5d38048e9030954a907ccbfb3fda204529067d3c024ba48d7e2dd329fe3dbc8
deleted: sha256:f1bd8b6a3b8bdec68c72a23e896e5278842da2a903c5b5fb090dd88f25edc676
deleted: sha256:529e9f64ea6c0660f0ee3dd29c98367d85ed90563d7767137113565703dceadc
deleted: sha256:83aa18cbe4a99007ea67f47ce6bec0756273775dac527ebaf8a0c224193f2b95
deleted: sha256:9a05df6eae3b21d822cd02d428c88c05ab67d5e8d11014884742ab22b7a44a7a
deleted: sha256:312ee2ba731589de51a59f88b05a206c45b3f07d7ac7c87eca16c0fef7475d74
deleted: sha256:0238a17903242951b944d01ab189a30ac237c715121f14565a2af8b078415d97
untagged: node:16
untagged: node@sha256:f77a1aef2da8d83e45ec990f45df50f1a286c5fe8bbfb8c6e4246c6389705c0b
deleted: sha256:1ddc7e4055fdb6f6bf31063b593befda814294f9f904b6ddfc21ab1513bafa8e
deleted: sha256:0322e47fe75b740cbf986b1750c334107718ab6d73bdbfedd71157cceccee639
deleted: sha256:da4a3a907432cf27b923ea845c1ed7560f9ce3dd23bc15bed71379e9d2b60eda
deleted: sha256:7d85e5185a181f323d84f04aea9584abb9049ae9e5dd43fab9974dfc3ce5dd89
deleted: sha256:eb298abd8219fcc019b01fa0cc7f57f925f7a4bfa904cabc9dc1de82d9d5ac7b
deleted: sha256:e384bd3e36d0d7d0e7ff836d2334856424095ef1874b02c092ed72edd07ab77e
deleted: sha256:c209cc51a67ea34ac3d3157f02e32a92020bf5b74a9d739f3fd15ee4b0ba675c
deleted: sha256:797b420c580e87b9747eaf2f4a0c545b56f0ae435ee01a4cda45da8e46840c70
deleted: sha256:9af5f53e8f62b46127a1e27488a8d501013df31fbb21d81fbce4cc55654fa910

Total reclaimed space: 3.148GB
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          0         0         0B        0B
Containers      0         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B

3GB解放されていることが分かります。私の環境では、これでとりあえずCIは動作するようになりました。

さいごに

急にディスク不足になる理由はよく分かっていません。とりあえずすぐCIを動かしたいときにご活用頂ければと思います。もっと良い方法がありましたら、@shuntaka_jpまでご連絡頂けると幸いです。